Image processing device, image processing method, and computer readable storage device

ABSTRACT

An image processing device includes: a candidate abnormal region determining unit that determines a candidate abnormal region from an image using a first determination criterion; a bubble region determining unit that determines a bubble region from the image; a bubble inside determining unit that determines whether the candidate abnormal region is present inside the bubble region based on a determination result of the bubble region; and an abnormal region determining unit that determines whether the candidate abnormal region is an abnormal region using a second determination criterion different from the first determination criterion when the candidate abnormal region is determined to be present inside the bubble region.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2011-229229, filed on Oct. 18, 2011, theentire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing device, an imageprocessing method, and a computer readable storage device fordetermining an abnormal region from an image obtained by imaging a lumenof a living body.

2. Description of the Related Art

In the related art, as image processing on images (hereinafter referredto as intra-luminal images or simply as images) obtained by imaging thelumen of a living body using a medical observation device such as anendoscope or a capsule endoscope, a technique of detecting an abnormalregion based on hue information of the image is disclosed in JapaneseLaid-open Patent Publication No. 2005-192880. More specifically, inJapanese Laid-open Patent Publication No. 2005-192880, the pixel valueof each pixel or a mean pixel value thereof is mapped onto a featurespace based on the color feature data, a normal mucosal cluster and anabnormal cluster are specified after performing clustering in thefeature space, and a pixel region belonging to the abnormal cluster isdetected as an abnormal region.

SUMMARY OF THE INVENTION

An image processing device according to an aspect of the inventionincludes: a candidate abnormal region determining unit that determines acandidate abnormal region from an image using a first determinationcriterion; a bubble region determining unit that determines a bubbleregion from the image; a bubble inside determining unit that determineswhether the candidate abnormal region is present inside the bubbleregion based on a determination result of the bubble region; and anabnormal region determining unit that determines whether the candidateabnormal region is an abnormal region using a second determinationcriterion different from the first determination criterion when thecandidate abnormal region is determined to be present inside the bubbleregion.

An image processing method according to another aspect of the inventionincludes: determining a candidate abnormal region from an image using afirst determination criterion; determining a bubble region from theimage; determining whether the candidate abnormal region is presentinside the bubble region based on the determination result of the bubbleregion; and determining whether the candidate abnormal region is anabnormal region using a second determination criterion different fromthe first determination criterion when the candidate abnormal region isdetermined to be present inside the bubble region.

A computer readable storage device according to still another aspect ofthe invention has an executable program stored thereon, wherein theprogram instructs a processor to perform: determining a candidateabnormal region from an image using a first determination criterion;determining a bubble region from the image; determining whether thecandidate abnormal region is present inside the bubble region based onthe determination result of the bubble region; and determining whetherthe candidate abnormal region is an abnormal region using a seconddetermination criterion different from the first determination criterionwhen the candidate abnormal region is determined to be present insidethe bubble region.

The above and other features, advantages and technical and industrialsignificance of this invention will be better understood by reading thefollowing detailed description of presently preferred embodiments of theinvention, when considered in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of an imageprocessing device according to a first embodiment of the invention;

FIG. 2 is a flowchart illustrating the operation of the image processingdevice illustrated in FIG. 1;

FIG. 3 is a schematic view illustrating a part of a processing targetimage as an example;

FIG. 4 is a flowchart illustrating a detailed process of determining anarc-shaped region;

FIG. 5 is a diagram explaining a method of determining a surroundingregion of a candidate abnormal region;

FIG. 6 is a flowchart illustrating a detailed process of calculating thearea of a bubble region in a surrounding region and a region in thevicinity of the surrounding region;

FIG. 7 is a block diagram illustrating the configuration of an imageprocessing device according to a second embodiment of the invention;

FIG. 8 is a flowchart illustrating the operation of the image processingdevice according to the second embodiment of the invention;

FIG. 9 is a flowchart illustrating a detailed process of determining acandidate abnormal region;

FIG. 10 is a diagram explaining a method of determining a bubble regiondetermination range;

FIG. 11 is a schematic view illustrating an arc-shaped region determinedfrom the determination range;

FIG. 12 is a diagram explaining a method of determining a candidatebubble region;

FIG. 13 is a flowchart illustrating a detailed process of extracting anarc-shaped candidate bubble region and an inner region;

FIG. 14 is a block diagram illustrating the configuration of an imageprocessing device according to a third embodiment of the invention;

FIG. 15 is a flowchart illustrating the operation of the imageprocessing device according to the third embodiment of the invention;

FIG. 16 is a schematic view explaining an image processing methodaccording to the third embodiment; and

FIG. 17 is a flowchart illustrating a detailed process of calculating afeature data based on a positional relation at each position of a bubbleregion.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter an image processing device, an image processing method, andan image processing program according to embodiments of the inventionwill be explained with reference to the drawings. The invention is notlimited to these embodiments. In the respective drawings, the sameportions are denoted by the same reference numerals.

In the following embodiments, as an example, a process on a series ofintra-luminal images (hereinafter simply referred to as images) obtainedby imaging the lumen of a subject in a time-sequential order using amedical observation device such as an endoscope or a capsule endoscopewill be explained. In the following description, the images subjected toimage processing are color images which have 256 pixel levels (pixelvalues) for each of the respective color components of R (red), G(green), and B (blue) at the respective pixel positions, for example.The invention is not limited to intra-luminal images but can be widelyapplied to a case of extracting a specific region from images acquiredusing other general image acquisition devices.

First Embodiment

FIG. 1 is a block diagram illustrating the configuration of an imageprocessing device according to a first embodiment of the invention. Animage processing device 1 illustrated in FIG. 1 includes a control unit10 that controls the operation of the entire image processing device 1,an image acquiring unit 20 that acquires image data corresponding to animage captured by a medical observation device, an input unit 30 thatreceives input signals input from an external device, a display unit 40that displays various screens, a storage unit 50 that stores the imagedata acquired by the image acquiring unit 20 and various programs, and aarithmetic unit 100 that executes predetermined image processing on theimage data.

The control unit 10 is implemented by hardware such as a CPU. Thecontrol unit 10 reads various programs stored in the storage unit 50 tothereby transmit instructions and data to the respective units of theimage processing device 1 in accordance with the image data input fromthe image acquiring unit 20 or an operation signal input from the inputunit 30 and control the operation of the entire image processing device1 in an integrated manner.

The image acquiring unit 20 is appropriately configured according to anaspect of a system that includes the medical observation device. Forexample, when the medical observation device is a capsule endoscope, anda portable recording medium is used in exchanging of image data with themedical observation device, the image acquiring unit 20 is configured asa reader device that detachably attaches the recording medium and readsimage data of the intra-luminal images stored in the recording medium.Moreover, when a server that stores image data of the intra-luminalimages captured by the medical observation device is provided, the imageacquiring unit 20 is configured as a communication device that isconnected to the server and performs data communication with the serverto acquire the image data of the intra-luminal images. Alternatively,the image acquiring unit 20 may be configured as an interface device orthe like that receives image signals from the medical observation devicevia a cable.

The input unit 30 is implemented as an input device such as, forexample, a keyboard, a mouse, a touch panel, or various switches, andoutputs received input signals to the control unit 10.

The display unit 40 is implemented as a display device such as an LCD oran EL display, and displays various screens including intra-luminalimages under control of the control unit 10.

The storage unit 50 is implemented as various IC memories called a ROMor a RAM such as a rewritable flash memory, an internal hard disk or ahard disk connected to a data communication terminal, or an informationstorage medium such as a CD-ROM, and a reading device thereof. Thestorage unit 50 stores a program for causing the image processing device1 to operate and causing the image processing device 1 to executevarious functions, and data or the like used during the execution of theprogram in addition to the image data of the intra-luminal imagesacquired by the image acquiring unit 20. Specifically, the storage unit50 stores an image processing program 51 for executing a process ofdetermining a candidate abnormal region from an image using a firstpredetermined criterion, determining a bubble region, and determiningwhether the candidate abnormal region is an abnormal region using asecond criterion when it is determined that the candidate abnormalregion is present within the bubble region. The storage unit 50 alsostores various determination criteria used during the execution of theimage processing program 51.

The arithmetic unit 100 is implemented by hardware such as a CPU. Thearithmetic unit 100 performs image processing on the image datacorresponding to the intra-luminal image by reading the image processingprogram 51 and performs various arithmetic processes for determining anabnormal region from an intra-luminal image.

Next, a detailed configuration of the arithmetic unit 100 will beexplained.

As illustrated in FIG. 1, the arithmetic unit 100 includes a candidateabnormal region determining unit 110 that determines a candidateabnormal region from an image using a first determination criterion, abubble region determining unit 120 that determines a bubble region fromthe image, a bubble inside determining unit 130 that determines whetherthe candidate abnormal region is present inside the bubble region basedon the determination result of the bubble region, and an abnormal regiondetermining unit 140 that determines whether the candidate abnormalregion is an abnormal region using a second determination criteriondifferent from the first determination criterion when the candidateabnormal region is determined to be present inside the bubble region.

The bubble region determining unit 120 includes a peripheral regiondetermining unit 121 that determines a region having the features of theperipheral region of the bubble region and an inner region determiningunit 122 that determines a region having the features of the innerregion of the bubble region. The peripheral region determining unit 121includes an arc-shaped region determining unit 121 a that determines anarc-shaped region which is the feature of the peripheral region of thebubble region. Moreover, the inner region determining unit 122 includesa halation region determining unit 122 a that determines a halationregion which is the feature of the inner region of the bubble region.

The bubble inside determining unit 130 includes a surrounding regiondetermining unit 131 that determines a surrounding region of thecandidate abnormal region and a surrounding region feature datacalculating unit 132 that calculates the feature data of the surroundingregion based on the determination result of the bubble region in thesurrounding region and a region in the vicinity of the surroundingregion. The surrounding region feature data calculating unit 132includes an area calculating unit 132 a that calculates the area of thebubble region in the surrounding region and the region in the vicinityof the surrounding region. More specifically, the area calculating unit132 a includes a contour extracting unit 132 a′ that extracts contourpixels of the surrounding region.

The abnormal region determining unit 140 includes a determinationcriterion switching unit 141 that switches the determination Criterionbased on whether the surrounding region of the candidate abnormal regionis present inside the bubble region.

Next, the operation of the image processing device 1 will be explained.FIG. 2 is a flowchart illustrating the operation of the image processingdevice 1.

First, in step S01, the image acquiring unit 20 acquires a series ofintra-luminal images obtained by imaging the lumen of a subject andstores the intra-luminal images in the storage unit 50. The arithmeticunit 100 sequentially reads image data corresponding to a processingtarget image from the storage unit 50. FIG. 3 is a schematic viewillustrating a part of the processing target image read by thearithmetic unit 100 as an example.

Subsequently, in step S02, the candidate abnormal region determiningunit 110 determines a candidate abnormal region based on the colorfeature data of the image. More specifically, the candidate abnormalregion determining unit 110 calculates a G/R value from the pixel valuesof the respective pixels that constitute an image and determines aregion in which the G/R value is smaller than a predetermineddetermination criterion value as a candidate abnormal region A10. Thatis, a region in which red colors are relatively strong (a region inwhich bleeding or reddening is suspicious) is extracted from the imageas the candidate abnormal region A10.

In step S03, the peripheral region determining unit 121 (the arc-shapedregion determining unit 121 a) determines an arc-shaped region in theimage. Although various methods can be used as a method of determiningthe arc-shaped region, in the first embodiment, a method disclosed inJapanese Laid-open Patent Publication No. 2007-313119 is used.

FIG. 4 is a flowchart illustrating a detailed process of step S03.First, in step S101, the arc-shaped region determining unit 121 acalculates a gradient intensity (G value) in the image. Subsequently, instep S102, a correlation value between the gradient intensity (G value)and an arc-shaped model created in advance is calculated. Further, instep S103, a region in which the correlation value between the gradientintensity and the arc-shaped model is equal to or greater than apredetermined threshold value is determined as an arc-shaped region A11(see FIG. 3). After that, the processing returns to a main routine.

In step S04, the inner region determining unit 122 (the halation regiondetermining unit 122 a) determines a halation region in the image.Specifically, a luminance value Y is calculated from the pixel values(RGB values) of the respective pixels using the following equation (1)(see Digital Image Processing, CG-ARTS Society, p. 299). Moreover, aregion in which the luminance value Y is equal to or greater than apredetermined threshold value is determined as a halation region A12(see FIG. 3).Y=0.3×R+0.59×G+0.11×B  (1)

In step S05, the bubble region determining unit 120 sets the arc-shapedregion A11 determined by the peripheral region determining unit 121 andthe halation region A12 determined by the inner region determining unit122 as the bubble region. In many cases, the bubble region includes boththe arc-shaped region A11 and the halation region A12.

In step S06, the surrounding region determining unit 131 determines thesurrounding region of the candidate abnormal region A10. Morespecifically, first, the surrounding region determining unit 131calculates the gravity center position G of the candidate abnormalregion A10 in the image as illustrated in FIG. 5. Moreover, a regionincluded in a circle of which the origin is located at the gravitycenter position G and which has a radius of r1 is extracted as asurrounding region A13. The value of the radius r1 may be apredetermined value or may be determined adaptively based on the area ofthe candidate abnormal region A10.

In step S07, the contour extracting unit 132 a′ executes a contourtracking process (see Digital Image Processing, CG-ARTS Society, p. 178)to extract a contour (outline) region A14 of the surrounding region A13.

In step S08, the area calculating unit 132 a calculates the area of thebubble region in the surrounding region A13 and a region in the vicinityof the surrounding region A13. For example, in the case of FIG. 3, thebubble region (the halation region) A12 is present in the surroundingregion A13, and the bubble region (the arc-shaped region) A11 is presentso as to overlap the contour region A14 of the surrounding region A13.The area calculating unit 132 a calculates the total areas of the bubbleregions A11 and A12.

FIG. 6 is a flowchart illustrating a detailed process of step S08. Instep S111, first, the area calculating unit 132 a calculates the totalarea S₁ of the bubble regions A11 and A12 in the surrounding region A13and the region in the vicinity of the surrounding region A13.Subsequently, in step S112, the area S₂ of a contour region (outline)A14 of the surrounding region A13 is calculated. Further, in step S113,the area S₁ of the bubble region is normalized by the area S₂ of thecontour region A14 of the surrounding region using the followingequation (2) to calculate a normalized bubble region area S.S=S ₁ /S ₂  (2)

In the above equation (2), the area S₁ may be divided by the area of thesurrounding region A13 instead of the area S₂ of the contour region A14,or the area S₁ may be divided by the sum of the areas of the surroundingregion A13 and the contour region A14.

After that, the processing returns to the main routine.

In step S09, the surrounding region feature data calculating unit 132uses the normalized bubble region area S as the feature data of thesurrounding region A13. Incidentally, the value itself of the area S₁ ofthe bubble regions A11 and A12 or the determination result (1 forpresent and 0 for absent) on the presence of a bubble region in thesurrounding region A13 and a region in the vicinity of the surroundingregion A13 may be used as the feature data of the surrounding regionA13.

In step S10, the bubble inside determining unit 130 determines whetherthe normalized bubble region area S in the surrounding region A13 of thecandidate abnormal region A10 is equal to or greater than apredetermined threshold value (predetermined value). When the normalizedbubble region area S is equal to or greater than the predetermined value(Yes in step S10), the bubble inside determining unit 130 determinesthat the candidate abnormal region A10 is inside the bubble region (stepS11). This is because the greater the area of the region having thefeatures of the bubble region in the surrounding region A13 and a regionin the vicinity of the surrounding region A13 is, the higher thepossibility of the candidate abnormal region A10 being an inner regionof the bubble region is. When the area S₁ of the bubble regions A11 andA12 or the presence of the bubble region in the surrounding region A13and the region in the vicinity of the surrounding region A13 is used asthe feature data of the surrounding region A13, the candidate abnormalregion A10 may be determined to be present inside the bubble region whenthe area S₁ is equal to or greater than a predetermined value or thebubble region is determined to be present.

Subsequently, in step S12, the determination criterion switching unit141 reads a determination criterion value created in advance from thestorage unit 50. The criterion value read at this time is smaller thanthe criterion value used by the candidate abnormal region determiningunit 110 in step S02. That is, the region (the abnormal region) which isdetermined to fall within the range defined by the criterion value readin step S12 is fewer than the region (the candidate abnormal region)which is determined to fall within the range defined by the criterionvalue in step S02. In other words, the criterion value read in step S12is stricter than the criterion value used in step S02.

In steps S13 to S15, the abnormal region determining unit 140 determineswhether the candidate abnormal region A10 is an abnormal region based onthe determination criterion read in step S12. Specifically, first, instep S13, the abnormal region, determining unit 140 calculates a meanvalue (G/R mean value) of the G/R value of the candidate abnormal regionA10. Subsequently, in step S14, the abnormal region determining unit 140determines whether the calculated G/R mean value is smaller than adetermination criterion. When the G/R mean value is determined to besmaller than the determination criterion (Yes in step S14), the abnormalregion determining unit 140 determines that the candidate abnormalregion A10 is an abnormal region such as a bleeding region or areddening region (step S15). In this case, a region in which the redcolors are stronger than the determination result in step S02 isdetermined as an abnormal region.

On the other hand, when the G/R mean value is equal to or greater thanthe determination criterion (No in step 314), the abnormal regiondetermining unit 140 determines that the candidate abnormal region A10is not an abnormal region (step S16).

Moreover, in step S10, when the normalized bubble region area S isdetermined to be smaller than a predetermined value (No in step S10),the candidate abnormal region A10 is determined not to be present insidethe bubble region (step S17). In this case, the candidate abnormalregion A10 is determined as an abnormal region according to thedetermination in step S02 (step S15).

As described above, according to the first embodiment, when a candidateabnormal region, which is determined based on a predetermined criterionvalue from an image, is determined to be present inside a bubble region,it is determined whether the candidate abnormal region is an abnormalregion in accordance with a different criterion value such that thecorresponding regions are fewer than that obtained when thepredetermined criterion value is used. Therefore, it is possible tosuppress a mucosal region inside bubbles from being erroneously detectedas an abnormal region.

First Modification Example

In the first embodiment described above, the G/R value and the luminancevalue Y have been calculated for each pixel, and the candidate abnormalregion and the halation region have been determined. However, an imagemay be divided into small regions, and the candidate abnormal region andthe halation region may be determined for each small region. In thiscase, a mean value in each small region of the G/R value and theluminance value Y of the respective pixels is used for the determinationprocess.

A process of dividing an image based on edge intensity is performed asfollows. First, the edge intensity of each of the respective pixelsincluded in a processing target image is calculated. In calculating theedge intensity, a known method such as a differential filtering processusing the Sobel filter may be used. Subsequently, the image is dividedinto multiple edge regions using the ridges of the edge intensities asboundaries. More specifically, an edge intensity image which uses therespective pixels edge intensities as pixel values is created, and agradient direction of the edge intensities in the pixels of the edgeintensity image is acquired. In this case, the gradient direction is setin a direction where the value of the edge intensity decreases.Moreover, pixels having the minimum value which the respective pixelsreach when moving along the gradient direction are searched, and theimage is divided so that the pixels at the starting point which havereached the adjacent pixels having the minimum value are included in thesame region (see WO 2006/080239 A).

As another image division method, an existing method such as a watershedalgorithm can be also used (see Luc Vincent and Pierre Soille,“Watersheds in digital spaces: An efficient algorithm based on immersionsimulations”, IEEE Transactions on Pattern Analysis and MachineIntelligence, Vol. 13, No. 6, pp. 583-598, June 1991).

According to the first modification example, since the candidateabnormal region and the halation region are determined based on thefeature data of each of the small regions including multiple pixels, itis possible to perform the determination process by reflecting thefeatures of each small region and to improve the computation speed.

Second Modification Example

In the first embodiment, it is determined whether the candidate abnormalregion is inside the bubble region using the value obtained bynormalizing the total area of the bubble region (the halation region)present in the surrounding region of the candidate abnormal region andthe bubble region (the arc-shaped region) near the contour region of thesurrounding region by the area of the contour region as the featuredata. However, it is not always necessary to extract the contour regionof the surrounding region. For example, the determination may beperformed using the total area of the halation region and the arc-shapedregion present in the surrounding region and the region in the vicinityof the surrounding region or a value obtained by normalizing the totalarea by the area of the surrounding region as the feature data.Alternatively, the determination may be performed based on whether apredetermined amount of the arc-shaped region is present in thesurrounding region and the region in the vicinity of the surroundingregion (for example, the total area of the arc-shaped region or the sumof the lengths of the arcs is equal to or greater than a predeterminedthreshold value).

Second Embodiment

Next, a second embodiment of the invention will be explained.

FIG. 7 is a block diagram illustrating the configuration of an imageprocessing device according to the second embodiment. As illustrated inFIG. 7, an image processing device 2 according to the second embodimentincludes an arithmetic unit 200 which includes a candidate abnormalregion determining unit 210 that determines a candidate abnormal regionfrom an image, a bubble region determining unit 220 that determines abubble region from the image, a bubble inside determining unit 230 thatdetermines whether the candidate abnormal region is present inside thebubble region based on the determination result of the bubble region,and an abnormal region determining unit 240 that determines whether thecandidate abnormal region is an abnormal region. The configuration ofthe image processing device 2 other than the arithmetic unit 200 is thesame as that illustrated in FIG. 1.

The bubble region determining unit 220 includes a peripheral regiondetermining unit 221 that determines a region having the features of theperipheral region of the bubble region and a determination rangedetermining unit 222 that determines a determination range where thebubble region in the image is determined based on the determinationresult of the candidate abnormal region. The peripheral regiondetermining unit 221 includes an arc-shaped region determining unit 221a that determines an arc-shaped region.

The bubble inside determining unit 230 includes a surrounding regiondetermining unit 231 that determines the surrounding region of thecandidate abnormal region and a surrounding region feature determiningunit 232 that determines the features of the bubble region in thesurrounding region based on the determination result of the bubbleregion in the surrounding region and the region in the vicinity of thesurrounding region.

More specifically, the surrounding region feature determining unit 232includes an arc-shaped bubble region determining unit 232 a thatdetermines whether the bubble region in the surrounding region and theregion in the vicinity of the surrounding region includes an arc-shapedregion and an arc-shaped inner region determining unit 232 b thatdetermines whether the surrounding region is an inner region of thearc-shaped region.

Next, the operation of the image processing device according to thesecond embodiment will be explained. FIG. 8 is a flowchart illustratingthe operation of the image processing device according to the secondembodiment.

First, in step S21, the arithmetic unit 200 acquires image datacorresponding to a processing target image. The detailed process of stepS21 is the same as that of step S01 of the first embodiment.

Subsequently, in step S22, the candidate abnormal region determiningunit 210 determines a candidate abnormal region based on a color featuredata.

FIG. 9 is a flowchart illustrating a detailed process of determining thecandidate abnormal region. In step S201, the candidate abnormal regiondetermining unit 210 calculates the G/R value from the pixel values ofthe respective pixels that constitute the image. In this case, similarlyto the first modification example, the mean value of the G/R values maybe calculated for each of the small regions that are obtained bydividing the image.

In step S202, the candidate abnormal region determining unit 210determines whether each of the calculated G/R values is smaller than afirst criterion value set in advance. When a region in which the G/Rvalue is smaller than the first criterion value is detected (Yes in stepS202), the region is determined as an abnormal region such as a bleedingregion or a reddening region (step S203).

On the other hand, with respect to the region in which the G/R value isequal to or greater than the first criterion value (No in step S202),the candidate abnormal region determining unit 210 determines whetherthe G/R value in the region is smaller than a second criterion value setin advance (second criterion value>first criterion value) (step S204).When the region in which the G/R value is smaller than the secondcriterion value is detected (Yes in step S204), the region is determinedas a candidate abnormal region which is likely to be an abnormal regionbut with low confidence (step S205). The region in which the G/R valueis equal to or greater than the second criterion value (No in step S204)is determined neither as an abnormal region nor a candidate abnormalregion. After that, the processing returns to the main routine.

In step S23, the determination range determining unit 222 determines adetermination range where the bubble region in the image is determined.Specifically, as illustrated in FIG. 10, the gravity center position Gof a candidate abnormal region A20 determined from the image iscalculated, and a region included in a circle of which the origin islocated at the gravity center position G and which has a radius of r2 isdetermined as a determination range A21. The value of the radius r2 maybe a predetermined value set in advance and may be determined adaptivelybased on the area of the candidate abnormal region A20.

In step S24, the peripheral region determining unit 221 (the arc-shapedregion determining unit 221 a) determines an arc-shaped region in thedetermination range A21. A detailed process of a method of determiningthe arc-shaped region is the same as that described with reference toFIG. 4 in step S03 of the first embodiment. FIG. 11 illustrates anexample of arc-shaped regions A22 to A26 determined in this way.

In step S25, the bubble region determining unit 220 sets the arc-shapedregions A22 to A26 determined in step S24 as candidate bubble regions.

In step S26, the surrounding region determining unit 231 determines thesurrounding region of the candidate abnormal region A20. A detailedprocess of step S26 is the same as that described in step S06 of thefirst embodiment. FIG. 12 illustrates a surrounding region A27determined in step S26.

In step S27, the arc-shaped bubble region determining unit 232 aextracts an arc-shaped candidate bubble region from the candidate bubbleregions (arc-shaped regions) A22 to A26 in the surrounding region A27 ofthe candidate abnormal region A20 and a region in the vicinity of thesurrounding region A27 and extracts an arc-shaped inner region.

FIG. 13 is a flowchart illustrating a detailed process of extracting anarc-shaped candidate bubble region and an arc-shaped inner region.

First, in step S211, the arc-shaped bubble region determining unit 232 aextracts a candidate bubble region in the surrounding region A27 of thecandidate abnormal region A20 and a region in the vicinity of thesurrounding region A27. In the case of FIG. 12, the candidate bubbleregions A22 to A24 and A26 are extracted from the candidate bubbleregions A22 to A26 set in step S25. Subsequently, in step S212, thearc-shaped bubble region determining unit 232 a calculates a correlationvalue between the extracted candidate bubble regions A22 to A24 and A26and an arc-shaped model created in advance. In step S213, a candidatebubble region in which the correlation value with the arc-shaped modelis equal to or greater than a predetermined threshold value isdetermined as an arc-shaped candidate bubble region. For example, in thecase of FIG. 12, the candidate bubble regions A22 to A24 are determinedas an arc-shaped candidate bubble region. Further, in step S214, thearc-shaped bubble region determining unit 232 a extracts an inner regionA28 of the candidate bubble regions A22 to A24 determined as anarc-shaped region. After that, the processing returns to the mainroutine.

In step S28, the arc-shaped inner region determining unit 232 bdetermines whether the arc-shaped inner region A28 extracted in step S27or the surrounding region A27 of the candidate abnormal region A20 is aninner region of an arc-shaped region.

When the surrounding region A27 of the candidate abnormal region A20 isan inner region of an arc-shaped region (Yes in step S28), the bubbleinside determining unit 230 determines that the surrounding region A27is inside a bubble region (step S29). In this case, in step S30, theabnormal region determining unit 240 determines that the candidateabnormal region A20 in the surrounding region A27 is not an abnormalregion.

On the other hand, when the surrounding region A27 of the candidateabnormal region A20 is not an inner region of an arc-shaped region (Noin step S28), the bubble inside determining unit 230 determines that thesurrounding region A27 is not inside the bubble region (step S31). Inthis case, in step S32, the abnormal region determining unit 240determines that the candidate abnormal region A20 in the surroundingregion A27 is an abnormal region.

As described above, according to the second embodiment, first, a regionin which the color feature data is smaller than a first criterion valueis determined as an abnormal region. With respect to only a region inwhich the color feature data is between the first criterion value andthe second criterion value and which is likely to be an abnormal regionwith low confidence, it is determined whether the region is an abnormalregion using a bubble region. Therefore, it is possible to improve theefficiency of the computation process.

Third Embodiment

Next, a third embodiment of the invention will be explained.

FIG. 14 is a block diagram illustrating the configuration of an imageprocessing device according to the third embodiment. As illustrated inFIG. 14, an image processing device 3 according to the third embodimentincludes an arithmetic unit 300 which includes the candidate abnormalregion determining unit 110, the bubble region determining unit 120, abubble inside determining unit 310 that determines whether the candidateabnormal region is present inside the bubble region based on thedetermination result of the bubble region, and an abnormal regiondetermining unit 320 that determines whether the candidate abnormalregion is an abnormal region based on the determination result of thebubble inside determining unit 310. The configuration and the operationof the candidate abnormal region determining unit 110 and the bubbleregion determining unit 120 are the same as those described in the firstembodiment. Moreover, the configuration of the image processing device 3other than the arithmetic unit 300 is the same as that illustrated inFIG. 1.

The bubble inside determining unit 310 includes a surrounding regiondetermining unit 311 that determines the surrounding region of thecandidate abnormal region and a surrounding region feature datacalculating unit 312 that calculates a feature data based on thesurrounding region. The surrounding region feature data calculating unit312 includes a positional relation-based feature data calculating unit312 a that calculates a feature data based on a positional relation ofthe bubble region in the surrounding region and the region in thevicinity of the surrounding region. The positional relation-basedfeature data calculating unit 312 a includes a distance calculating unit312 a-1 that calculates the distance between bubble regions and a bubbleregion positional relation-based feature data calculating unit 312 a-2that calculates the feature data based on a positional relation betweenthe arc-shaped region and the halation region.

The abnormal region determining unit 320 includes a determinationcriterion creating unit 321 that adaptively creates a determinationcriterion based on the information on the surrounding region of thecandidate abnormal region.

Next, the operation of the image processing device according to thethird embodiment will be explained. FIG. 15 is a flowchart illustratingthe operation of the image processing device according to the thirdembodiment.

In FIG. 15, the operations of steps S41 to S46 correspond to theoperations of steps S01 to S06 illustrated in FIG. 2. FIG. 16illustrates a candidate abnormal region A30, an arc-shaped region A31, ahalation region A32, and a surrounding region A33 which are determinedfrom the image in steps S41 to S46.

In step S47, the bubble region positional relation-based feature datacalculating unit 312 a-2 determines whether both the arc-shaped regionA31 and the halation region A32 which constitute the bubble region aremixedly present in the surrounding region A33 and the region in thevicinity of the surrounding region A33. Here, in the bubble region, ahalation is generally observed in the inner side of an arc-shaped regionthat constitutes a portion having an approximately circular shape(including a shape similar to a circle such as an ellipse). Thus, aregion where both a portion determined as the arc-shaped region A31 anda portion determined as the halation region A32 are mixedly present canbe determined not to be the bubble region.

When both the arc-shaped region A31 and the halation region A32 are notmixedly present in the surrounding region A33 and the region in thevicinity of the surrounding region A33 (No in step S47), the positionalrelation-based feature data calculating unit 312 a calculates thefeature data based on the positional relation in each portion of thebubble region (step S48).

FIG. 17 is a flowchart illustrating a detailed process of calculatingthe feature data based on the positional relation in each portion of thebubble region. First, in step S401, the bubble region positionalrelation-based feature data calculating unit (hereinafter, simplyreferred to as a feature data calculating unit) 312 a-2 calculates thegravity center position G of the candidate abnormal region A30 in theimage. Subsequently, in step S402, the feature data calculating unit 312a-2 calculates the mean distance C from the gravity center position G ofthe candidate abnormal region A30 to the respective arc-shaped regionsA31 present in the surrounding region A33 and the region in the vicinityof the surrounding region A33. Moreover, in step S403, the feature datacalculating unit 312 a-2 calculates a mean distance H from the gravitycenter position G of the candidate abnormal region A30 to the respectivehalation regions A32 present in the surrounding region A33 and theregion in the vicinity of the surrounding region A33. Further, thefeature data calculating unit 312 a-2 calculates a difference D betweenthe mean distance C to the arc-shaped region A31 and the mean distance Hto the halation region A32 using the following equation (3) (step S404).D=C−H  (3)

In step S405, the positional relation-based feature data calculatingunit 312 a sets the difference D calculated in this way as a featuredata based on the positional relation in each portion of the bubbleregion in the surrounding region A33 and the region in the vicinity ofthe surrounding region A33.

In step S49, the surrounding region feature data calculating unit 312sets the feature data (the difference D) based on the positionalrelation in each portion of the bubble region as a feature data in thesurrounding region A33.

In step S50, the bubble inside determining unit 310 determines whetherthe feature data (namely, the difference D) in the surrounding regionA33 is equal to or greater than zero.

When the feature data in the surrounding region A33 is equal to orgreater than zero (Yes in step S50), the bubble inside determining unit310 determines that the candidate abnormal region 730 is inside thebubble region (step S51). This is because the fact that the difference Dis equal to or greater than zero means that the halation region A32 canbe determined to be present inside the arc-shaped region A31.

Subsequently, in step S52, the determination criterion creating unit 321calculates the mean value (G/R mean value) of the G/R values in portionsof the surrounding region A33 other than the candidate abnormal regionA30 and the halation region A32.

In steps S53 to S55, the abnormal region determining unit 320 determineswhether the candidate abnormal region A30 is an abnormal region usingthe G/R mean value calculated in step S52 as a determination criterion.Specifically, first, in step S53, the G/R mean value in the respectivecandidate abnormal regions A30 is calculated. Moreover, in step S54, itis determined whether a difference D_(AB) between the G/R mean value inthe surrounding region A33 and the G/R mean value in the candidateabnormal region A30 is equal to or greater than a predeterminedthreshold value (predetermined value). When the difference D_(AB) isequal to or greater than the predetermined value (Yes in step S54), theabnormal region determining unit 320 determines that the candidateabnormal region A30 is an abnormal region (step S55). On the other hand,when the difference D_(AB) is smaller than the predetermined value (Noin step S54), the abnormal region determining unit 320 determines thatthe candidate abnormal region A30 is not an abnormal region (step S56).

Moreover, when it is determined in step S47 that both the arc-shapedregion and the halation region are mixedly present in the surroundingregion A33 and the region in the vicinity of the surrounding region A33(Yes in step S47), the bubble inside determining unit 310 determinesthat the candidate abnormal region is not inside the bubble region (stepS57). In this case, in step S55, the abnormal region determining unit320 determines that the candidate abnormal region A30 is an abnormalregion.

Moreover, even when the feature data D is smaller than zero (No in stepS50), the bubble inside determining unit 310 determines that thecandidate abnormal region is not inside the bubble region (step S57).

As described above, according to the third embodiment, since it isdetermined whether the surrounding region of the candidate abnormalregion is present inside the bubble region based on the positionalrelation of the portions that constitute the bubble region, it ispossible to improve the detection accuracy of the bubble region.

Third Modification Example

In the first to third embodiments described above, although the G/Rvalue has been used as the color feature data which is used indetermining the candidate abnormal region, various color feature datessuch as the respective RGR values, relative values (B/G values or thelike) of the respective RGB values, brightness and color differencecalculated by YCbCr conversion, or hue, saturation, and lightnesscalculated by HSI conversion can be used. For example, when the B/Gvalue is used as the color feature data, it is easy to determine alesion of a region covered by the bile that is yellow. Here, B and Gcomponents of an illumination light that illuminates the lumen of asubject are absorbed in a red region by approximately the same amount.Thus, there is not a great difference between the B and G values in manyregions of the lumen and a bleeding region. However, since the amount ofabsorption of the B component increases in the bile, the B valuedecreases in the bile region, and it becomes easy to detect a change inthe G value. Thus, a region in the image in which the B/G value is equalto or greater than a predetermined value can be determined as a lesion,and a region in which the B/G value is smaller than the predeterminedvalue can be determined as a region other than the lesion.

According to the first to third embodiments and the modificationexamples thereof, when the candidate abnormal region determined based onthe first determination criterion is determined to be present inside thebubble region, it is determined whether the candidate abnormal region isan abnormal region using the second determination criterion differentfrom the first determination criterion. Thus, it is possible to suppressa mucosal region inside bubbles from being erroneously detected as anabnormal region.

The image processing device according to the first to third embodimentsand the modification examples thereof can be implemented by an imageprocessing program recorded on a recording medium being executed by acomputer system such as a personal computer or a workstation. Moreover,the computer system may be used in a state of being connected to anapparatus such as another computer system or a server via a local areanetwork, a wide area network (LAN/WAN) or a public line such as theInternet. In this case, the image processing device according to thefirst to third embodiments may acquire image data of the intra-luminalimages via these networks, output image processing results to variousoutput apparatuses (a viewer, a printer, and the like) connected viathese networks, and store the image processing results in a storagedevice (a recording medium, a reading device thereof, and the like)connected via these networks.

The invention is not limited to the first to third embodiments and themodification examples thereof, but various inventions can be formed byappropriately combining multiple constituent components disclosed in therespective embodiments and modification examples. For example, someconstituent components may be removed from all constituent componentsillustrated in the respective embodiments and modification examples, andconstituent components illustrated in other embodiments and modificationexamples may be appropriately combined.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

What is claimed is:
 1. An image processing device for an endoscope,comprising: a processor comprising hardware, the processor beingconfigured to implement: a candidate abnormal region determining unitthat calculates a color feature data by using pixel values of anintra-luminal image captured by the endoscope and determines a candidateabnormal region from the intra-luminal image by comparing the calculatedcolor feature data with a first determination criterion; a bubble regiondetecting unit that detects a bubble region from the intra-luminalimage, wherein the bubble region is a region on which a bubblecomprising a body of gas within a liquid is captured in theintra-luminal image; a bubble inside determining unit that determineswhether the candidate abnormal region is present inside the detectedbubble region based on a detection result of the bubble region; and anabnormal region determining unit that determines whether the candidateabnormal region is an abnormal region using an amount of red in thecalculated color feature data and a second determination criterion lessinclusive with respect to the amount of red than the first determinationcriterion when the candidate abnormal region is determined to be presentinside the detected bubble region; wherein portions in the intra-luminalimage that are determined to be abnormal region by satisfying the seconddetermination criterion are fewer than portions in the intra-luminalimage that are determined to be the candidate abnormal region bysatisfying the first determination criterion; and wherein the bubbleregion detecting unit detects, as the bubble region, a region having anarc-shaped region and a halation region from the intra-luminal image. 2.The image processing device according to claim 1, wherein the abnormalregion determining unit includes a determination criterion switchingunit that switches to the second determination criterion based on adetermination result of the bubble inside determining unit.
 3. The imageprocessing device according to claim 1, wherein the abnormal regiondetermining unit includes a determination criterion creating unit thatadaptively creates the second determination criterion based oninformation on a surrounding region of the candidate abnormal region. 4.The image processing device according to claim 3, wherein portions inthe intra-luminal image that are determined to be the abnormal region bysatisfying the second determination criterion are fewer than portions inthe intra-luminal image that are determined to be the abnormal region bysatisfying the first determination criterion.
 5. The image processingdevice according to claim 1, wherein the bubble region detecting unitincludes a determination range determining unit that determines adetermination range where the bubble region in the intra-luminal imageis detected based on the determination result of the candidate abnormalregion.
 6. The image processing device according to claim 1, wherein thebubble region detecting unit includes a peripheral region determiningunit that determines a region which has features of a peripheral regionof the bubble region.
 7. The image processing device according to claim6, wherein the peripheral region determining unit includes an arc-shapedregion determining unit that determines the arc-shaped region.
 8. Theimage processing device according to claim 1, wherein the bubble regiondetecting unit includes an inner region determining unit that determinesa region which has features of an inner region of the bubble region. 9.The image processing device according to claim 8, wherein the innerregion determining unit includes a halation region determining unit thatdetermines the halation region.
 10. The image processing deviceaccording to claim 1, wherein the bubble inside determining unitincludes a surrounding region determining unit that determines asurrounding region surrounding the candidate abnormal region, and asurrounding region feature data calculating unit that calculates asurrounding region feature data that is a feature data of thesurrounding region based on the detection result of the bubble region inthe surrounding region and a region in the vicinity of a contour of thesurrounding region among the detection results of the bubble region bythe bubble region detecting unit.
 11. The image processing deviceaccording to claim 10, wherein the surrounding region feature datacalculating unit includes an area calculating unit that calculates thearea of the bubble region in the surrounding region and the region inthe vicinity of the surrounding region.
 12. The image processing deviceaccording to claim 11, wherein the area calculating unit includes acontour extracting unit that extracts contour pixels of the surroundingregion, calculates the area of the contour pixels, and calculates anormalized area of the area of the bubble region based on the area ofthe contour pixels.
 13. The image processing device according to claim10, wherein the surrounding region feature data calculating unitincludes a positional relation-based feature data calculating unit thatcalculates a feature data based on a positional relation of multipleportions that configure the bubble region in the surrounding region andthe region in the vicinity of the surrounding region.
 14. The imageprocessing device according to claim 13, wherein the positionalrelation-based feature data calculating unit includes a distancecalculating unit that calculates the distance between the multipleportions in the surrounding region and the region in the vicinity of thesurrounding region and the candidate abnormal region.
 15. The imageprocessing device according to claim 13, wherein the bubble regiondetecting unit includes a peripheral region determining unit thatdetermines a region which has the features of the peripheral region ofthe bubble region included in the multiple portions; and an inner regiondetermining unit that determines a region which has the features of theinner region of the bubble region included in the multiple portions, andthe positional relation-based feature data calculating unit includes abubble region positional relation-based feature data calculating unitthat calculates a feature data based on a positional relation betweenthe region which has the features of the peripheral region and theregion which has the features of the inner region.
 16. The imageprocessing device according to claim 1, wherein the bubble insidedetermining unit includes a surrounding region determining unit thatdetermines a surrounding region of the candidate abnormal region; and asurrounding region feature determining unit that determines the featuresof the bubble region in the surrounding region based on the detectionresult of the bubble region in the surrounding region and the region inthe vicinity of the surrounding region among the detection results ofthe bubble region by the bubble region detecting unit.
 17. The imageprocessing device according to claim 16, wherein the surrounding regionfeature determining unit includes an arc-shaped bubble regiondetermining unit that determines whether the bubble region in thesurrounding region and the region in the vicinity of the surroundingregion includes the arc-shaped region; and an arc-shaped inner regiondetermining unit that determines whether the surrounding region is aninner region of the arc-shaped region.
 18. An image processing methodfor an endoscope using a processor and a memory storing computerreadable instructions that, when executed by the processor, implementthe steps comprising: calculating a color feature data by using pixelvalues of an intra-luminal image captured by the endoscope anddetermining a candidate abnormal region from the intra-luminal image bycomparing the calculated color feature data with a first determinationcriterion; detecting a bubble region from the intra-luminal image,wherein the bubble region is a region on which a bubble comprising abody of gas within a liquid is captured in the intra-luminal image;determining whether the candidate abnormal region is present inside thedetected bubble region based on a detection result of the bubble region;and determining whether the candidate abnormal region is an abnormalregion using an amount of red in the calculated color feature data and asecond determination criterion less inclusive with respect to the amountof red than the first determination criterion when the candidateabnormal region is determined to be present inside the detected bubbleregion; wherein portions in the intra-luminal image that are determinedto be the abnormal region by satisfying the second determinationcriterion are fewer than portions in the intra-luminal image that aredetermined to be the candidate abnormal region by satisfying the firstdetermination criterion; and wherein the detecting the bubble regiondetects, as the bubble region, a region having an arc-shaped region anda halation region from the intra-luminal image.
 19. A non-transitorycomputer readable storage device with an executable program storedthereon, wherein the program instructs a processor to perform for anendoscope: calculating a color feature data by using pixel values of anintra-luminal image captured by the endoscope and determining acandidate abnormal region from the intra-luminal image by comparing thecalculated color feature data with a first determination criterion;detecting a bubble region from the intra-luminal image, wherein thebubble region is a region on which a bubble comprising a body of gaswithin a liquid is captured in the intra-luminal image; determiningwhether the candidate abnormal region is present inside the detectedbubble region based on a detection result of the bubble region; anddetermining whether the candidate abnormal region is an abnormal regionusing an amount of red in the calculated color feature data and a seconddetermination criterion less inclusive with respect to the amount of redthan the first determination criterion when the candidate abnormalregion is determined to be present inside the detected bubble region;wherein portions in the intra-luminal image that are determined to bethe abnormal region by satisfying the second determination criterion arefewer than portions in the intra-luminal image that are determined to bethe candidate abnormal region by satisfying the first determinationcriterion; and wherein the detecting the bubble region detects, as thebubble region, a region having an arc-shaped region and a halationregion from the intra-luminal image.